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Abstract 


The  U.S.  Army  Strategic  Software  Improvement  Program  (ASSBP)  is  a  multi-year  effort  to 
improve  the  way  the  Army  acquires  software-intensive  systems.  As  part  of  the  ASSIP,  the 
Carnegie  Mellon®  Software  Engineering  Institute  interviewed  the  Army’s  Program  Executive 
Officers  (PEOs)  to  get  their  perspectives  on  the  state  of  Army  acquisition  of  software¬ 
intensive  systems.  The  PEOs  are  senior  acquisition  professionals  with  a  wealth  of  experience 
in  Army  acquisition  and  the  attempts  to  improve  it  in  the  past. 

This  report  documents  the  results  of  the  PEO  interviews.  The  PEOs  identified  shortcomings 
and  some  interesting  potential  solutions  in  various  aspects  of  the  acquisition  function,  includ¬ 
ing  skills  and  training,  policy,  the  acquisition  organization,  metrics,  process,  commercial  off- 
the-shelf  (COTS)  products,  and  facilities  and  tools. 
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1  Introduction 


In  late  2002,  the  Carnegie  Mellon®  Software  Engineering  Institute  (SEI)  entered  into  a  strate¬ 
gic  agreement  with  the  assistant  secretary  of  the  Army  for  acquisition,  logistics,  and  technol¬ 
ogy  (ASA(ALT)).  This  partnership,  known  as  the  Army  Strategic  Software  Improvement 
Program  (ASSIP),  seeks  to  improve  the  Army’s  techniques  for  acquiring  systems  with  high 
software  content,  called  software-intensive  systems1,  or  SIS. 

This  special  report  relates  the  results  of  interviews  with  the  Army’s  Program  Executive  Offi¬ 
cers  (PEOs),  which  the  SEI  conducted  as  part  of  the  ASSIP.  The  SEI  is  using  the  results  to 
help  guide  the  ASSIP  improvement  initiatives.  The  intent  of  this  report  is  to  stimulate  open 
discussions  about  SIS  acquisition  issues  by  sharing  the  information  with  the  acquisition  func¬ 
tions  of  the  other  services,  other  government  agencies,  and  industry. 

In  order  to  understand  these  results,  a  brief  review  of  the  ASSIP  and  of  Army  acquisition  will 
be  helpful. 


1 .1  The  Army  Strategic  Software  Improvement  Pro¬ 
gram 

The  ASSIP  is  a  multi-year  effort  targeted  at  dramatically  improving  the  way  the  Army  ac¬ 
quires  software-intensive  systems.  The  ASA(ALT)  proactively  initiated  the  ASSIP  in  2002  as 
a  means  of  helping  the  Army  respond  to  the  challenges  of  developing  systems  that  are  in¬ 
creasingly  dependent  on  software.  Later,  when  Congress  included  Section  804  in  the  Na¬ 
tional  Defense  Authorization  Act  for  Fiscal  Year  2003,  which  required  the  services  to  develop 
plans  for  their  software  acquisitions  [PL  02],  the  ASSIP  was  easily  shaped  to  address  those 
requirements  as  well. 


Organizationally,  there  are  two  main  bodies  involved  in  the  ASSIP.  The  Senior  Steering 
Group  (SSG),  composed  of  the  PEOs,  the  MILDEP,  and  the  Director  of  the  SEI,  provides 
overall  guidance  to  the  effort.  The  ASSIP  Action  Group  (AAG),  consisting  of  representatives 
from  each  of  the  PEOs  and  from  the  Army  Software  Engineering  Centers,  as  well  as  SEI 


1  According  to  the  Defense  Acquisition  University  (DAU),  a  software-intensive  system  is  one 
in  which  software  represents  the  largest  segment  in  one  or  more  of  the  following  criteria:  system 
development  cost,  system  development  risk,  system  functionality,  or  development  time  [DAU  03]. 
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technical  staff  members,  develops  and  implements  improvement  strategies.  The  ASSIP  is  a 
partnership,  and  the  SSG  and  AAG  are  co-chaired  by  the  Army  and  the  SEI.  In  addition,  the 
SEI  offers  expert  guidance  on  software  acquisition  and  process  issues,  provides  secretariat 
services  to  the  SSG  and  AAG  and  acts  as  a  catalyst  for  change.  As  shown  in  Figure  1,  the 
ASSIP  is  predicated  on  the  idea  that  better  acquisition  practices  will  lead  to  better  systems 
and  overall  results. 


Figure  1:  ASSIP  Uses  Improved  Acquisition  Processes  to  Yield  Better  Results 

As  with  any  improvement  effort,  for  the  ASSIP  to  be  successful,  an  understanding  of  the 
baseline  state  of  Army  acquisition  was  needed.  While  there  was  plenty  of  anecdotal  evidence 
and  speculation  about  the  challenges  to  successful  SIS  acquisition,  there  was  very  little  hard 
evidence  on  which  to  build  improvement.  The  SEI  set  about  gathering  the  necessary  evi¬ 
dence  through  several  means:  a  survey  of  Army  acquisition  managers,  direct  engagements 
with  several  critical  Army  programs  using  a  technique  called  Benchmarking  for  Improve¬ 
ment,  and  the  PEO  interviews.  The  first  two  methods  are  described  briefly  below,  while  the 
PEO  interviews  are  the  subject  of  the  body  of  this  report. 


1 .1 .1  Survey  of  Acquisition  Managers 

The  SEI  made  an  initial  attempt  at  characterizing  the  state  of  Army  acquisition  through  a  sur¬ 
vey  of  Army  acquisition  managers.  While  the  survey,  conducted  in  2003,  proved  useful  to  set 
SEI  expectations  about  the  range  of  potential  problems,  it  did  not  provide  reliable  or  suffi¬ 
ciently  detailed  information  on  which  to  base  improvement  strategies.  Instructions  to  the  tar¬ 
get  audience  were  not  adequately  communicated  to  achieve  consistency  of  respondents. 

Many  managers  delegated  responsibility  for  completing  the  survey  to  staff  members,  whose 
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backgrounds  varied  widely.  Interested  readers  may  refer  to  the  report  Artny  Strategic  Soft¬ 
ware  Improvement  Program  (ASSIP)  Survey  of  Army  Acquisition  Managers  for  a  detailed 
discussion  of  the  survey  results  and  their  limitations  [Kasunic  04], 


1 .1 .2  Benchmarking  for  Improvement 

Benchmarking  for  Improvement  (BFT)  formed  the  backbone  of  the  ASSIP  data  gathering  ef¬ 
forts.  This  technique  employs  a  series  of  structured  interviews  with  program  office  personnel 
(similar  to  many  program  assessment  techniques),  but  the  focus  is  on  discovery  of  the  proc¬ 
esses  in  use  rather  than  on  compliance  with  some  standard.  BFI  seeks  to  elicit  those  proc¬ 
esses  employed  by  a  program,  using  a  model  as  a  guideline,  instead  of  rating  program  office 
processes  against  a  model.  Benchmarking  began  in  mid-2003  and  continued  throughout 
2004. 


Although  BFI  results  for  individual  programs  remain  the  property  of  the  respective  program 
managers,  the  SEI  has  aggregated  the  results  in  a  non-attributable  manner  to  foster  analysis. 
The  results  are  remarkably  consistent  with  the  results  from  the  PEO  interviews.  Refer  to  the 
report  Results  of  the  Benchmarking  for  Improvement  Effort  in  the  ASSIP2  for  detailed  discus¬ 
sion  of  the  BFI  results. 


1 .2  The  Structure  of  Army  Acquisition 

This  section  provides  only  high-level  insight  into  the  structure  of  Army  acquisition.  For  de¬ 
tails  about  organization  structure  and  individual  responsibilities,  refer  to  the  document  Army 
Regulation  70-1:  Army  Acquisition  Policy  [HQDA03]. 

The  Honorable  Claude  M.  Bolton,  Jr.  is  the  ASA(ALT)  and  the  Army  Acquisition  Executive 
(AAE).  The  highest  ranking  uniformed  official  in  Army  acquisition  is  Mr.  Bolton’s  Military 
Deputy  (MILDEP),  a  position  currently  held  by  Lieutenant  General  Joseph  L.  Yakovac,  Jr. 
The  ASA(ALT)  organization  has  the  responsibility  for  Army  materiel  acquisition.  In  his  role, 
Mr.  Bolton  appoints,  manages,  and  evaluates  the  Program  Executive  Officers  and  Program 
Managers  (PMs). 


Generally,  the  Army  clusters  its  acquisition  programs  under  one  of  12  program  executive  of¬ 
fices,  each  headed  by  a  senior  acquisition  professional  referred  to  as  a  Program  Executive 
Officer  (PEO).  The  PEOs  provide  oversight  for  a  group  of  related  programs.  Table  1  shows 
the  program  executive  offices  at  the  time  of  these  interviews. 


2  Capell,  Peter  &  Keeler,  Kristi  L.  Results  of  the  Benchmarking  for  Improvement  Effort  in  ASSIP. 
Pittsburgh,  PA:  Software  Engineering  Institute,  Carnegie  Mellon  University,  to  be  published. 


CMU/SEI-2005-SR-002 


3 


Table  1:  The  Program  Executive  Offices  (as  of  December  2004) 


Air,  Space  and  Missile  Defense  (ASMD) 
Ammunition  (Ammo) 

Aviation  (AVN) 

Chemical  and  Biological  Defense  (CBD) 

Combat  Support  and  Combat  Service  Support 
(CS&CSS) _ 

Command,  Control  and  Communications  Tactical 
(C3T) 


Enterprise  Information  Systems  (EIS) 


Ground  Combat  Systems  (GCS) 


Intelligence,  Electronic  Warfare  and  Sensors  (IEW&S) 


Soldier 


Simulation,  Training,  and  Instrumentation  (STRI) 


Tactical  Missiles  (TM) 


In  some  cases,  a  program  manager  may  report  directly  to  the  AAE  without  an  intervening 
PEO.  At  the  time  of  the  interviews,  two  such  direct  reporting  PMs,  PM  Joint  Tactical  Radio 
System  (JTRS)  and  PM  Unit  of  Action  (UA),  existed.  Figure  2  depicts  the  organization  in 
simplified  terms.  This  graphic  is  by  no  means  a  complete  representation;  it  only  attempts  to 
convey  the  relative  position  of  the  PEOs  within  the  Army  acquisition  hierarchy. 


Figure  2:  Simplified  View  of  Army  Acquisition  Structure 

1 .3  About  Army  PEOs 

PEOs  can  be  senior  military  officers  or  civilian  members  of  the  government’s  senior  execu¬ 
tive  service  (SES).  They  serve  as  the  materiel  developers,  responsible  for  programmatics  and 
the  various  aspects  of  planning  and  budgeting  required  to  steward  their  assigned  programs 
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through  the  applicable  milestones.  In  addition  to  their  many  other  duties,  PEOs  are  also  re¬ 
sponsible  for  technical  and  functional  integration  across  their  assigned  programs  [HQDA  03]. 

The  DoD  Acquisition  Career  Development  Program  [DoD  95]  defines  PEO  qualifications. 
Generally,  before  assignment,  PEOs  must  have  completed  specific  levels  of  training,  must 
have  been  assigned  to  a  PM  or  deputy  PM  position  during  their  careers,  and  must  have  had  at 
least  10  years  of  acquisition  experience  (with  4  of  those  being  in  a  critical  acquisition  posi¬ 
tion).  The  position  of  PEO  itself  is  a  critical  acquisition  position,  which  levies  additional  re¬ 
quirements. 
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2  Method  Description 


The  SEI  conducted  interviews  with  the  PEOs  from  April  2004  through  December  2004.  In¬ 
terview  teams  consisted  of  two  SEI  technical  staff  members,  with  one  member  acting  as  the 
primary  interviewer  and  the  other  acting  as  the  primary  recorder.  To  provide  consistency 
across  the  interviews,  only  two  individuals  acted  as  the  primary  interviewer  during  the  course 
of  the  interviews. 


The  interview  questions  were  divided  into  two  sets:  primary  and  secondary.  The  set  of  pri¬ 
mary  questions  dealt  with  the  PEOs’  opinions  about  Army  acquisition  overall  and  the  ways  in 
which  the  ASSIP  could  help.  The  set  of  secondary  questions  dealt  mainly  with  specific 
ASSIP  activities  and  activities  in  each  PEO’s  office.  Appendix  B  contains  the  interview 
questions. 


The  AAG  members  from  each  PEO  received  all  the  questions  as  read-ahead  material  to  facili¬ 
tate  scheduling  the  interviews.  This  allowed  the  PEOs  to  become  familiar  with  the  discussion 
materials  and  to  decide  if  they  needed  any  support  from  their  staffs  during  the  interviews.  In 
addition,  the  AAG  representatives  pre-briefed  their  respective  PEOs  on  current  ASSIP  status 
prior  to  the  interview  meeting. 


Each  interview  was  conducted  individually;  that  is,  only  one  PEO  was  interviewed  during 
each  session.  Although  the  PEOs  typically  had  their  AAG  representative(s)  with  them,  the 
PEOs  themselves  answered  all  of  the  questions,  with  only  occasional  embellishment  from 
their  staff.  Interviews  were  conducted  in  the  PEOs’  offices. 


To  accommodate  the  PEOs’  busy  schedules,  interview  sessions  were  limited  to  90  minutes. 

In  a  few  instances,  interviews  ran  longer  than  the  allotted  time,  but  only  if  the  PEO  wanted  to 
continue.  One  interview  began  late  and  ran  less  than  the  allotted  time  due  to  the  PEO’s  other 
commitments.  Generally,  interviewers  omitted  the  secondary  questions  if  time  ran  short. 

The  interviewers  sometimes  omitted  some  primary  questions  if  the  answers  to  other  questions 
sufficiently  covered  the  topic. 


While  the  goal  of  the  interviews  was  to  obtain  opinions  about  the  acquisition  of  software- 
intensive  systems  specifically,  interviewers  did  not  interrupt  the  discussions  if  they  touched 
on  broader  acquisition  issues,  and  nearly  all  of  the  interviews  did.  Software  is  but  one  of 
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many  interrelated  components  of  an  acquisition  program,  and  understanding  the  whole  pic¬ 
ture  is  critical  to  addressing  systemic  problems  rather  than  individual  symptoms. 


Not  all  interviewees  held  the  position  of  PEO.  The  SEI  also  interviewed  two  deputy  PEOs 
and  one  direct  reporting  PM.  For  ease  of  discussion,  the  entire  group  of  interviewees  is  re¬ 
ferred  to  as  PEOs  in  this  report.  Readers  may  find  demographic  information  about  the  inter¬ 
view  group  in  Appendix  C. 
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3  The  PEO  Perspective 


Each  of  the  PEOs  expressed  a  variety  of  concerns  about  the  state  of  Army  acquisition.  Al¬ 
though  they  each  had  specific  problems  within  their  respective  domains,  they  also  identified 
some  systemic  issues  as  a  group.  This  report  addresses  only  the  systemic  problems,  but  if 
available,  the  SEI  provided  additional  information,  references,  and/or  contacts  to  assist  PEOs 
with  domain-specific  issues.  In  addition  to  identifying  problems,  some  PEOs  offered  ideas 
for  potential  solutions.  While  the  SEI  maintains  the  confidentiality  of  individual  respondents, 
this  section  of  the  report  includes  non-attributed  quotes  that  punctuate  the  findings.  Note  that 
the  SEI  did  not  attempt  to  verify  the  opinions  or  assertions  expressed  herein. 

Upon  reviewing  the  interview  responses,  several  broad  categories  of  concern  emerged: 

•  Skills  and  Training 

•  Policy 

•  The  Acquisition  Organization 

•  Metrics 

•  Process 

•  Commercial  Off-the-Shelf  (COTS)  Products 

•  Facilities  and  Tools 

The  following  sections  discuss  each  of  these  categories  in  more  detail.  The  order  of  discus¬ 
sion  is  not  relevant  or  significant,  and  readers  should  not  ascribe  priority  or  importance  to  any 
of  the  categories  based  on  the  order. 


3.1  Skills  and  Training 

Perhaps  the  most  relevant  comment  regarding  skills  and  training  was  “I  don’t  know  if  the 
Army  possesses  enough  smart  buyer  insight  for  software.”  Another  PEO  echoed  the  senti¬ 
ment:  “The  government. .  .needs  to  be  a  smart  buyer.”  Most  of  the  PEOs  acknowledged  a 
mismatch  between  the  software  expertise  of  contractors  and  the  program  offices. 
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Although  some  PEOs  were  comfortable  with  the  level  of  software  knowledge  possessed  by 
their  PMs\  others  said  that  their  PMs  were  not  as  knowledgeable  about  software  as  they 
needed  to  be.  The  PEOs  who  felt  that  their  PMs’  software  comprehension  was  lacking  ex¬ 
pressed  a  range  of  needs,  from  “knowing  what  they  don’t  know”  to  “knowing  what  their  staff 
should  know”  to  specific  technical  deficiencies  such  as,  requirements  management,  software 
architecture,  and  so  forth. 

Making  training  opportunities  available  to  PMs  is  difficult,  due  to  the  heavy  demands  on  their 
time.  One  PEO  said,  “We  really  can’t  send  them  back  to  class  all  the  time.”  One  PEO 
faulted  the  Acquisition  Corps  for  not  producing  technically  skilled  PMs  from  the  start. 

“We’re  creating  generalists  out  of  specialists,  moving  people  around  instead  of  letting  them 
build  [technical]  capability.”  This  particular  PEO  went  on  to  say  that  Army  acquisition  is 
lacking  technical  talent  because  of  the  overly  generalist  orientation  of  the  Acquisition  Corps. 

Knowledge  needs  extend  up  to  the  PEO  level,  as  well.  One  PEO  noted,  “I  know  all  about  [a 
particular  weapon  system],”  meaning  the  physical/mechanical  aspects  of  it,  “I  don’t  have  ex¬ 
pertise  in  software  development.”  Some  PEOs  were  seeking  guidance  at  an  even  more  basic 
level.  What  does  the  Army  judge  to  be  the  necessary  insights  for  PEOs?  Is  there  a  consistent 
set  of  knowledge  that  PEOs  should  possess?  These  two  questions  point  to  a  fundamental  un¬ 
certainty  on  the  part  of  some  PEOs. 


3.2  Policy 

Although  none  of  the  primary  interview  questions  asked  specifically  about  it,  the  topic  of 
policy  came  up  in  every  interview.  The  issues  expressed  included  policies  at  the  Department 
of  Army-level  and  higher.  Generally,  PEOs  used  the  term  “policy”  to  refer  to  a  range  of  re¬ 
lated  guiding  documents,  including  directives,  guidance,  law,  regulations,  and  so  forth.  For 
the  purposes  of  this  study,  no  differentiation  is  made. 

This  section  of  the  report  addresses  two  specific  policy  areas  frequently  identified  during  the 
interviews  -  software  blocking  and  interoperability.  The  remainder  of  this  section  discusses 
the  general  policy  concerns  that  the  PEOs  identified. 


3.2.1  Software  Blocking 

Virtually  without  exception,  the  PEOs  singled-out  the  Army’s  Software  Blocking  policy  as  an 
example  of  bad  policy.  The  software  blocking  policy  attempts  to  create  a  known  set  of  com¬ 
patible  software  (developed  by  disparate  programs)  for  tactical  Command,  Control,  Commu- 


3  In  the  context  of  subordinates  to  the  PEOs,  “PM”  can  refer  to  Program  Manager,  Project  Manager, 
or  Product  Manager.  While  each  of  these  positions  has  meaning  within  Army  acquisition,  the 
differences  are  not  germane  to  the  discussion,  and  therefore  are  not  differentiated. 
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nications.  Computers,  Intelligence,  Surveillance,  and  Reconnaissance  (C4ISR)  systems.  Un¬ 
der  this  policy,  each  compatible  “block”  would  be  fielded  at  regular,  18-month  intervals. 
Prior  to  institution  of  this  policy,  programs  fielded  their  C4ISR  systems  according  to  their 
individual  schedules,  which  sometimes  resulted  in  an  inability  to  share  C4ISR  data  across 
platforms,  due  to  incompatibilities. 


The  PEOs  felt  that  the  software  blocking  policy  is  poorly  defined  and  executed,  as  reflected 
by  statements  like  “It’s  killing  the  Army.”  One  PEO  also  noted  that  one  of  the  primary  pro¬ 
ponents  of  the  policy  “doesn’t  have  a  hammer,”  implying  that  this  supporter  lacks  the  clout  to 
truly  enforce  it. 

Another  complaint  about  software  blocking  is  that  it  does  not  consider  program  constraints, 
such  as  the  need  to  field  software  updates  and  patches  more  frequently  than  every  18  months. 
This  was  a  particular  concern  for  PEOs  whose  programs  make  extensive  use  of  commercial- 
off-the-shelf  (COTS)  software,  since  COTS  patches  tend  to  be  released  frequently,  especially 
for  security. 

Finally,  the  PEOs  indicated  that  software  blocking  ignores  important  technical  considera¬ 
tions.  For  example,  one  PEO  pointed  out  that  the  policy  does  not  consider  hardware  com¬ 
patibility  issues.  As  a  result,  although  C4ISR  software  releases  may  be  coordinated  across 
systems,  individual  systems  can  experience  software-hardware  incompatibilities  that  degrade 
or  disable  essential  functions. 

3.2.2  Interoperability 

Interoperability  was  another  popular  topic  among  the  PEOs.  In  general,  they  recognize  the 
need  and  importance  of  developing  systems  that  can  interoperate  on  the  battlefield.  How¬ 
ever,  the  vagueness  of  the  term  itself  causes  many  of  them  to  doubt  its  achievability.  The 
statement  “It’s  just  a  cost  line  to  the  PMs;  no  one  understands  what  the  directive  means,” 
sums  up  the  situation.  The  definition  of  interoperability  within  the  Army  (and  within  the 
DoD)  appears  to  be  context  sensitive,  meaning  different  things  to  different  people  depending 
on  their  perspectives.  Without  clear  definition,  interoperability  becomes  an  abstract,  daunting 
challenge.  As  one  PEO  asked,  “How  do  we  bound  interoperability?”  The  question  extends 
beyond  the  operational  considerations  of  “who  do  I  talk  to”  into  doctrinal  concerns  of  “who 
should  I  be  able  to  talk  to,  and  when,”  especially  in  joint  service,  interagency,  and  multina¬ 
tional  operations. 

Adding  to,  or  perhaps  at  the  root  of,  the  confusion  is  a  lack  of  a  central  authority  on  the  sub¬ 
ject.  Programs,  working  with  their  user  representatives  and  the  other  organizations  that  gen¬ 
erate  requirements,  are  able  to  identify  and  negotiate  interfaces  to  other  programs  on  an  ad 
hoc  basis.  However,  several  PEOs  identified  the  need  for  “one  trail  boss,”  that  is,  a  central 
authority  with  the  broader  perspective  on  interoperability.  One  PEO  suggested  that  this  au- 
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thority  should  reside  at  the  Department  of  Army  or  Joint  Services  level.  Others  thought  that 
the  assistant  secretary  of  defense  for  networks  and  information  integration  (ASD(NII))  should 
assume  responsibility.  Said  one  PEO  regarding  interoperability  standards  and  enforcement, 
“ASD(NII)  must  set  the  standards.” 

As  one  PEO  noted,  real  technical  barriers  make  achieving  interoperability  difficult,  such  as 
limited  radio  capability,  dissimilar  networks  and  data  formats,  dissimilar  data  processing 
across  platforms  and  services,  to  name  a  few.  However,  interoperability  is  not  simply  a  tech¬ 
nical  issue.  PEOs  and  PMs  must  also  determine  how  to  align  schedules,  deliverables,  and 
other  logistics  across  organizations.  Cultural  resistance  is  also  a  factor.  Relating  interopera¬ 
bility  to  joint  warfighting,  one  PEO  admitted,  “OSD  believes  [the  services]  don’t  want  to  be 
joint,  and  there  is  some  truth  to  that.” 


3.2.3  Broad  Policy  Concerns 

In  addition  to  the  concerns  raised  about  software  blocking  and  interoperability,  the  PEOs 
made  some  observations  on  policy  matters  in  general. 

3.2.3.1  Policy  Applicability  and  Volume 

The  applicability  of  policy  sparked  concern.  According  to  the  PEOs,  policies  are  frequently 
developed  with  acquisition  category  I  (ACAT  I)4  programs  in  mind,  yet  all  levels  of  ACAT 
programs  must  address  the  policies  ACAT  II  and  ACAT  III  programs  receive  no  guidance  for 
tailoring  policies  for  use  at  their  respective  levels. 

The  sheer  volume  of  policy  causes  confusion.  ‘There’s  too  much  policy,”  was  a  common 
complaint.  Perhaps  even  more  surprising,  was  the  claim  that  determining  which  version  of  a 
given  policy  is  in  effect  is  sometimes  difficult. 

3.2.3.2  Policy  Contradictions 

Many  PEOs  observed  that  contradictions  between  DoD  policies  and  Army  policies  is  a  prob¬ 
lem.  One  PEO  offered  a  solution:  have  policy  writers  in  the  office  of  the  undersecretary  of 
defense  for  acquisition,  technology,  and  logistics  (OUSD(AT&L))  coordinate  with  their  ser¬ 
vice  counterparts  to  provide  consistent  acquisition  policies  throughout  the  DoD. 


4  Programs  are  classified  into  different  acquisition  categories,  primarily  based  on  various  dollar 
values.  The  ACAT  is  used,  among  other  things,  to  determine  the  level  of  the  Milestone  Decision 
Authority  (MDA).  ACAT  I  programs  are  major  defense  acquisition  programs.  Programs  can  also 
be  designated  as  ACAT  I  by  the  MDA  if  the  MDA  has  a  special  interest.  ACAT  II  programs  are 
those  that  do  not  meet  the  ACAT  I  criteria  but  are  still  considered  major  systems,  or  those  programs 
so  designated  by  the  MDA.  ACAT  III  programs  are  those  that  do  not  meet  the  ACAT  I  or  ACAT  II 
criteria  [HQDA03]. 
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Apparently,  the  clarity  of  policies  leaves  much  to  be  desired.  One  PEO  suggested,  “It  would 
be  great  if  policies  came  with  examples.”  A  number  of  PEOs  pointed  out  that  expectations 
regarding  implementation  are  not  always  clear.  One  PEO  described  the  bring  me  a  rock  sce¬ 
nario5,  where  PMs  submit  a  document  intended  to  address  a  given  policy  several  times,  each 
time  believing  that  they  have  completed  the  task,  before  the  approver  finally  accepts  the 
document  as  sufficient. 

3.2.3.3  Policy  Development 

“Policy  and  directive  writers  appear  to  lack  tactical  experience,”  is  how  one  PEO  character¬ 
ized  the  gap  between  the  authors  and  implementers  of  policy.  A  couple  of  PEOs  even  implied 
that  policy  writing  was  a  way  to  keep  Pentagon  staffers  busy,  an  indictment  not  only  of  the 
qualifications  of  policy  authors,  but  of  the  entire  bureaucracy.  Many  PEOs  expressed  the  de¬ 
sire  to  see  the  implementation  level  (PEOs  and  PMs)  have  a  greater  say  in  the  development 
of  policy,  feeling  that  this  would  help  address  the  concerns  they  have  identified.  In  the  words 
of  one  PEO,  ‘The  more  policy  derivation  done  at  the  PEO/PM  level,  the  better.” 

3.2.3.4  Policy  Implementation  Gap 

Almost  all  PEOs  noted  that  policies  are  frequently  imposed  without  the  funding  to  support 
them,  forcing  programs  to  siphon  money  from  their  budgets  to  comply.  Some  PEOs  also 
questioned  whether  compliance  is  really  being  checked.  Given  these  practices,  a  harsh  de¬ 
scription  of  reality  emerged:  if  customers  do  not  care  about  a  given  policy,  system  function¬ 
ality  and  other  funded  priorities  supersede  policy  implementation. 


3.3  The  Acquisition  Organization 

The  PEOs  expressed  concerns  in  a  variety  of  areas  related  to  the  way  the  Army  is  organized 
to  perform  the  acquisition  function. 

3.3.1  Organizational  Efficiency 

Some  PEOs  bemoaned  what  they  perceived  as  a  lack  of  efficiency  in  Army  acquisition. 
“We’re  too  bureaucratic,”  said  one  PEO,  who  went  on  to  say  that,  “we  need  better  oversight, 
not  less  oversight.”  Inefficiency  in  oversight  is  a  large  contributor  to  overall  inefficiency  in 
the  Army  acquisition  organization.  The  resultant  slowness  of  the  acquisition  system  is  frus¬ 
trating  for  users.  “[Users]  would  rather  develop  their  own  solutions”  than  approach  the  ac¬ 
quisition  community,  because  they  can  get  the  solutions  faster.  The  increasing  availability  of 
COTS  products  only  serves  to  exacerbate  the  problem.  Noted  one  PEO,  commanders  go  to 
stores  and  to  trade  shows,  and  they  see  new  features  and  capabilities,  and  they  want  those 


5  “Bring  me  a  rock”  is  a  classic  story  of  frustration.  Each  time  you  bring  me  a  rock,  I  tell  you  it  is  the 
wrong  rock,  without  providing  any  guidance  as  to  what  the  right  rock  might  be. 
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new  capabilities  to  enhance  fielded  systems.  When  the  acquisition  process  proves  unable  to 
support  them,  they  “go  out  and  buy  the  off-the-shelf  stuff  [themselves]”,  and  “that’s  not 
right.” 

Some  PEOs  identified  staffers  as  a  significant  source  of  oversight  bureaucracy.  In  order  to 
get  required  “sign-offs”  (document  approvals,  permission  to  proceed,  etc.),  PMs  and  PEOs 
must  go  through  multiple  staffers  who  apply  inconsistent  and  uncoordinated  guidance.  As  a 
solution,  one  PEO  suggested  that  for  each  approver,  there  should  be  one  (and  only  one)  “des¬ 
ignated  representative  that  we  could  work  with.”  Another  PEO  said  bluntly,  “Look  at  the 
leadership  group.  Who  makes  decisions?  Who  adds  value?  Remove  the  rest  and  re¬ 
form/restructure  the  group.” 


3.3.2  The  Role  of  Software  Engineering  Centers 

Interviews  explicitly  covered  the  use  of  the  Army’s  Software  Engineering  Centers6  (SECs) 
because  the  SEI  wanted  to  understand  how  their  expertise  was  being  leveraged  to  supplement 
program  office  skills.  The  response  from  the  PEOs  indicated  that  SEC  expertise  was  used 
sporadically.  Some  PEOs  benefited  from  having  an  SEC  nearby,  or  even  co-located  with, 
their  program  offices.  For  others,  SECs  were  not  as  conveniently  placed. 

Surprisingly,  some  PEOs  chose  not  to  use  SEC  services  even  when  they  were  conveniently 
available  because  they  perceived  the  SECs  to  be  “weak”  in  certain  tactical  domains.  This 
perception  caused  PEOs  to  pursue  other  options  for  support  when  tactical  domain  experience 
was  as  important  as  (and  sometimes  more  important  than)  the  software  expertise. 

All  of  the  PEOs  appeared  to  make  good  use  of  the  resources  available  to  them,  whether  from 
the  SECs,  other  commands,  or  industry.  The  PEOs  who  were  unable  or  unwilling  to  use  SEC 
support  were  usually  able  to  find  the  help  they  needed  from  support  contractor  organizations 
or  from  organizations  with  software  expertise  in  the  other  services. 


Although  not  explicitly  asked,  a  couple  of  PEOs  brought  up  the  subject  of  software  mainte¬ 
nance,  which  is  typically  handled  by  the  SECs  after  a  system  is  fielded.  One  of  those  PEOs 
was  emphatic,  saying,  “ASSIP  needs  to  look  at  how  software  is  supported.”  Apparently,  as 
the  nature  of  acquisition  changes  from  individual  systems  to  “systems  of  systems,”  the  exist¬ 
ing  post-production  software  support  (PPSS)  concept  is  viewed  as  being  inadequate. 


6  The  Army  has  several  Software  Engineering  Centers,  operated  as  part  of  the  Army  Materiel 
Command,  that  provide  an  array  of  software  lifecycle  support  for  the  Army,  as  well  as  for  other 
services. 
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3.3.3  The  Role  of  Contractors 


Some  PEOs  took  issue  with  the  practice  of  relying  heavily  on  prime  contractors.  One  PEO 
acknowledged  that  acquisition  reform  sought  to  achieve  “better,  faster,  cheaper”  by  allowing 
industry  do  the  work  (referring  to  past  initiatives  like  Total  System  Performance  Responsibil¬ 
ity  [TSPR],  etc.),  but  it  hasn’t  worked  out.  “[Industry  has]  different  goals”  than  DoD.  An¬ 
other  said,  “[The  Lead  System  Integrator  (LSI)  approach]  is  costing  the  services  millions  of 
dollars.”  He  continued,  “[DoD  needs]  to  be  deliberate”  about  when  to  use,  and  when  not  to 
use,  an  LSI. 


3.3.4  Command  Continuity 

One  frequently  mentioned  organizational  problem  was  the  lack  of  command  continuity 
within  programs.  Currently,  the  Army  reassigns  its  military  personnel  (including  program 
managers)  after  two-three  years  on  the  job.  Since  most  programs  last  considerably  longer, 
there  is  a  break  in  leadership  continuity  whenever  the  current  PM  “rotates”  out  and  a  new  PM 
comes  in.  As  is  the  case  for  anyone  starting  a  new  job,  the  new  PM  has  a  learning  curve  to 
scale  before  becoming  truly  effective  in  the  position.  Some  programs  have  civilian  PMs,  and 
they  tend  to  stay  in  their  positions  longer,  although  they  are  still  subject  to  rotational  assign¬ 
ments.  As  one  PEO  noted,  civilian  PMs  used  to  be  the  norm.  “We  used  to  take  proven  man¬ 
agers  and  assign  them  until  a  program  was  in  good  shape.  We  used  to  have  civilians  as  PMs 
for  five-eight  years  with  the  military  as  deputies.”  But  the  solution  is  not  necessarily  to  make 
the  job  of  PM  a  civilian  position  across  the  board.  As  another  PEO  put  it,  “[Civilians]  can’t 
go  up  against  a  green-suiter,”  expressing  doubt  that  Army  culture  would  allow  civilian  PMs 
to  be  effective  in  dealing  with  uniformed  peers  and  superiors. 


The  PEOs  had  some  interesting  thoughts  on  how  to  combat  the  problem  of  continuity.  For 
instance,  one  suggested  developing  a  “scorecard”  for  the  life  of  a  program,  so  that  a  PM 
would  continue  to  bear  some  responsibility  for  program  success  even  after  moving  to  a  new 
assignment. 


Another  PEO  suggested  that  the  duration  of  an  assignment  should  be  flexible.  “A  two-year 
assignment  expectation  on  an  ACAT I  program  does  not  make  sense.  The  PM  cannot  get 
good  at  the  job  in  two  years.”  Instead,  a  PM  should  remain  in  a  position  until  achieving  some 
measurable  accomplishment.  The  PM  should  be  held  accountable  for  the  assignment;  it 
should  not  be  just  a  matter  of  getting  the  program  to  the  next  milestone  and  “leaving  the  real 
problems  for  the  next  guy.”  Another  PEO  said  it  slightly  differently:  “PMs  aren’t  around 
long  enough  to  see  the  consequences  of  their  actions.” 
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3.3.5  Lifecycle  Management 

In  August  2004,  the  ASA(ALT)  signed  a  memorandum  of  agreement  with  the  commander  of 
the  Army  Materiel  Command  (AMC)  regarding  lifecycle  management.7  The  details  of  the 
implementation  were  still  in  progress  at  the  time  many  of  the  PEO  interviews  were  con¬ 
ducted,  so  the  PEOs  were  unable  to  offer  any  specific  opinions  about  it.  However,  many  (but 
not  all)  expressed  some  optimism  that  the  agreement  would  be  helpful.  Said  one,  “[Maybe  it] 
will  put  someone  in  charge  for  the  life  of  a  program.”  However,  at  least  one  PEO  disagreed, 
saying  the  arrangement  “further  dilutes  a  PM’s  focus.”  This  particular  PEO  went  on  to  say 
that  the  agreement  creates  two  lines  of  authority,  one  within  the  ASA(ALT)  organization  and 
another  within  AMC,  leaving  the  question,  “Who  do  you  report  to?” 


3.3.6  PM  Selection 

The  PEOs  had  a  great  deal  to  say  about  the  selection  process  for  PMs.  The  general  feeling 
was  that  the  process  does  not  work  well.  Currently,  candidate  PMs  are  allowed  to  include  a 
“preference  statement”  regarding  potential  duty  assignments,  including  preferred  domiciles. 
Some  of  the  PEOs  felt  that  the  process  routinely  ignores  the  preference  statement  during  as¬ 
signment  of  PM  positions. 

The  PEOs  offered  some  interesting  ideas  for  adjusting  the  selection  process.  Perhaps  the 
most  straightforward  idea  was  to  allow  the  PM  candidates  to  bid  for  positions  that  make  the 
most  sense  based  on  their  background  and  domicile  preferences.  In  other  words,  make  the 
applicant’s  preference  statement  meaningful  by  giving  it  more  weight  in  the  process.  The 
focus  should  be  on  getting  the  right  person  for  the  job. 

Another  idea  was  to  let  the  candidates  themselves  make  the  selections.  ‘These  are  well- 
trained  professionals.  Let  them  work  together  to  fill  the  positions,”  said  one  PEO.  The  can¬ 
didates  could  go  through  the  list,  select  the  jobs  they  want  (and  for  which  they  possess  the 
right  qualifications),  and  then  negotiate  among  themselves  to  determine  final  assignments. 

One  PEO  warned  against  putting  out  “take  it  or  leave  it”  job  assignments.  It  forces  PMs  to 
make  a  career  choice,  and  too  many  opt  to  leave  the  service  (for  higher  pay  in  industry)  rather 
than  take  an  assignment  that  is  not  in  their  preferred  location  or  is  a  poor  match  for  their 
background. 


7  Memorandum  of  Agreement  between  the  Assistant  Secretary  of  the  Army  for  Acquisition,  Logistics, 
and  Technology  and  the  Commander,  U.S.  Army  Materiel  Command,  dated  2  August  2004. 

Subject:  Life-Cycle  Management  (LCM)  Initiative. 
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3.3.7  Staffing 

Overall,  most  PEOs  expressed  some  concern  about  the  reduction  of  the  acquisition  workforce 
in  both  government  and  support  contractor  personnel.  One  PEO  summed  it  up  as,  “We  need 
a  cadre  of  people  with  expertise,  and  we  need  to  have  consultant  advice  available  as  well.” 
However,  as  discussed  in  Section  3.1,  finding  candidates  with  the  right  skills  seems  to  be  dif¬ 
ficult,  and  training  existing  staff  has  not  been  a  priority  in  some  organizations. 


3.3.8  Beyond  AL&T 

Some  PEOs  highlighted  difficulties  in  working  with  the  stakeholder  community  that  is  not 
part  of  the  ASA(ALT)  organization.  One  PEO  observed  a  degree  of  friction  between  Army 
acquisition  and  the  requirements  development  function  of  the  Army’s  Training  and  Doctrine 
Command  (TRADOC)8,  calling  it  “counter-productive”  to  the  goal  of  providing  the  best 
products  to  the  soldier.  However,  a  desire  to  overcome  the  friction  and  work  the  issues  is 
evident.  Said  one  PEO,  “We  should  reach  out  to  TRADOC”  to  develop  a  collaborative  un¬ 
derstanding  of  operational  and  technical  architectures.  This  statement  reflects  both  a  funda¬ 
mental  disconnect  between  acquisition  and  user  proponents  and  an  interest  in  working  to¬ 
gether  to  solve  the  problem. 

Despite  organizational  attempts  to  innovate,  the  bottom  line  is  that  there  is  no  incentive  to  try 
new  approaches.  “People  want  to  [try  new  things  to  improve  acquisition],  so  get  out  of  the 
way,”  as  one  PEO  put  it.  The  consistent  problem  is  that  stakeholder  organizations  are  not 
“bought  in”  to  changes.  Streamlining  within  the  acquisition  organization  quickly  meets 
“business  as  usual”  at  its  interfaces,  thus  canceling  out  the  streamlining  effect  and  leading  to 
frustration  all  around. 


3.4  Metrics 

The  PEOs  all  expressed  some  level  of  frustration  with  metrics,  particularly  as  applied  to  SIS. 
The  key  concern  is  understanding  when  they  (or  their  PMs)  need  to  be  worried  about  soft¬ 
ware  development.  Comments  such  as,  “We  have  metrics  on  contract.  Some  appear  to  be 
incorrect”  and  “We  don’t  have  the  right  insight  yet”  typified  the  frustration.  Even  a  PEO  who 
was  reasonably  comfortable  with  the  metrics  used  acknowledged  sometimes  being  caught  off 
guard  by  product  quality  problems  and  escalating  contractor  costs.  ‘There  are  no  good  met¬ 
rics”  to  project  those  surprises. 

In  general,  two  schools  of  thought  emerged  regarding  software  metrics.  One  school  sought 
standardized  metrics,  and  felt  that  the  ASSIP  should  be  the  mechanism  to  provide  them.  The 


8  Among  other  things,  TRADOC  is  responsible  for  developing  requirements  for  systems  and 
providing  the  interface  between  program  offices  and  system  users. 
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other  felt  that  every  program  is  different  and  therefore  needs  individualized  metrics.  Note 
that  these  are  not  necessarily  mutually  exclusive  goals;  conceivably,  one  could  tailor  a  set  of 
metrics  for  the  specific  needs  of  individual  programs  from  a  broader  set  of  standard  metrics. 

Most  PEOs  reviewed  general  program  metrics  from  their  respective  programs  periodically 
(typically,  every  one-two  months),  relying  on  the  PMs  to  report  exceptional  information  more 
frequently  as  necessary.  While  several  PEOs  expressed  a  desire  to  avoid  burdening  PMs  un¬ 
necessarily,  they  also  indicated  a  desire  for  better  (as  opposed  to  more)  information.  Some  of 
the  data  desired  included  more  details  about  how  PMs  use  the  financial  resources  available  to 
them  and  better  insight  into  matrixed  personnel  usage  at  both  the  program  and  PEO  levels. 


3.5  Process 

The  PEOs  identified  a  lack  of  overall  process  discipline  within  Army  acquisition.  Referring 
to  the  lack  of  a  standard  process  for  managing  SIS,  one  PEO  noted,  “Everyone  does  it  differ¬ 
ently.”  A  few  PEOs  were  extremely  critical  of  the  Army’s  process  discipline,  indicating  that 
the  Army  is  too  willing  to  change  (or  worse,  subvert)  its  processes  if  following  those  proc¬ 
esses  would  lead  to  “unpopular”  decisions. 

Most  PEOs  acknowledged  that  process,  by  itself,  was  not  a  solution.  One  noted  that  the  best 
Army  acquisition  executives  “focused  on  mission  and  performance.”  Several  PEOs  pointed 
out  that  the  Army,  and  the  DoD  as  well,  has  changed  processes  in  the  past,  but  failed  to 
change  the  associated  behaviors.  The  result  of  these  paper-only  changes  is  that  the  new  proc¬ 
esses  have  not  been  any  more  effective  than  the  old  ones,  because  nothing  has  really  changed 
at  all.  Further,  each  failure  to  actually  effect  change  causes  the  people  who  might  benefit 
from  improvements  to  view  all  subsequent  attempts  with  suspicion,  ultimately  making  true 
change  more  difficult. 

Some  specific  process  areas  are  of  considerable  concern  to  the  PEOs.  Several  noted  require¬ 
ments  management  as  a  difficulty  (said  one,  “Requirements  management  kills  me”).  Some 
PEOs  have  the  added  complication  of  addressing  more  than  one  user  community  and,  there¬ 
fore,  multiple  sources  of  requirements.  Risk  management  was  also  acknowledged  as  a  trou¬ 
ble  spot  for  some. 

As  expected,  no  one  liked  the  unwieldy  process  that  must  be  followed  to  obtain  funding. 
However,  although  many  complained  about  the  legendarily  cumbersome  system  for  funding 
programs,  a  more  pressing  concern  was  what  happened  after  budgets  were  in  place.  The  ten¬ 
dency  to  shift  money  from  funded  programs  to  pay  for  other  expenses  makes  it  exceedingly 
difficult  for  PEOs  and  PMs  to  manage  their  budgets  and,  ultimately,  their  programs.  Noted 
one  PEO,  the  constant  changes  “cannot  help  but  cause  requirements  creep  and  system 
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thrash.”  This  presents  an  insidious  issue  for  all  of  DoD  acquisition,  and  has  many  sources 
within  each  service  department,  within  the  DoD,  and  within  Congress. 


3.6  Commercial-Off-the-Shelf  Products 

Beyond  software  blocking  implications,  several  PEOs  expressed  concerns  about  COTS  prod¬ 
ucts  (both  software  and  hardware)  and  their  increasing  presence  in  Army  systems.  Referring 
to  COTS  hardware,  one  PEO  stated,  “The  big  problem  with  COTS  is  that  later  you  can’t  get  a 
specific  component,”  due  to  obsolescence,  forcing  redesign  and  added  costs.  “COTS  compo¬ 
nents  drive  up  the  cost.” 


As  for  COTS  software,  one  PEO  noted  that  it  posed  both  technical  and  non-technical  chal¬ 
lenges.  “Although  COTS  software  does  a  great  deal,  it  often  doesn’t  meet  all  of  the  require¬ 
ments.”  He  went  on  to  say,  “Commanders  go  to  stores  and  to  trade  shows,  and  they  see  new 
features  and  capabilities,  and  they  want  those,  too.”  It  seems  the  need  for  expectations  man¬ 
agement  can  be  an  unanticipated  outcome  from  using  COTS.  Resisting  the  trend  to  incorpo¬ 
rate  COTS  as  a  routine  acquisition  strategy,  another  PEO  said  he  was  “not  an  advocate  of 
COTS  unless  it  makes  sense  in  the  context.”  In  other  words,  there  should  be  a  reasoned 
judgment  about  the  applicability  of  COTS  in  a  given  system,  rather  than  including  it  as  a  per¬ 
functory  cost  savings  attempt.  Finally,  another  PEO  noted  that  DAU  has  a  toolkit  that  helps 
managers  deal  with  COTS  products,  “but  they  must  still  re-leam  how  to  manage  COTS  and 
the  process  for  using  COTS.” 


3.7  Facilities  and  Tools 

Many  of  the  PEOs  have  geographically  dispersed  organizations.  As  with  any  such  organiza¬ 
tion,  communication  is  a  challenge.  PEOs  noted  secure  email,  Web,  and  video  teleconferenc¬ 
ing  access  as  being  critical  to  maintaining  communication  within  their  respective  organiza¬ 
tions  as  well  as  with  external  stakeholders.  However,  several  PEOs  have  some  part  of  their 
organizations  located  in  commercial  facilities,  making  it  difficult  to  get  the  necessary  secure 
access.  One  PEO  quoted  8-14  months  as  the  time  lag  for  installing  critical  secure  communi¬ 
cations  in  commercial  facilities.  This  delay  necessitates  finding  alternate  facilities  to  perform 
work,  attend  meetings,  and  so  on. 

There  were  several  complaints  about  some  of  the  automated  tools  that  the  PEOs  use  on  a 
regular  basis.  The  tools  do  not  yield  the  desired  time  savings.  Said  one  PEO,  the  tools  shift 
the  “burden  onto  supervisors  while  making  the  bureaucrat’s  job  easier.”  The  civilian  person¬ 
nel  and  travel  authorization  systems  were  identified  as  examples  of  aggravating  systems. 

One  PEO  said,  “It  takes  20  minutes  to  do  what  used  to  take  3  minutes”  using  old,  paper-based 
systems.  Another  comment,  “the  way  the  Army  handles  personnel  is  frustrating,”  illustrates 
concern  over  not  having  a  good  way  to  get  the  full  perspective  on  personnel.  As  a  solution, 
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one  PEO  commented  that  someone  in  ASA(ALT)  needs  to  look  at  all  these  systems  to  see  if 
there’s  truly  value  added. 
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4  Next  Steps 


This  report  documented  the  raw  results  of  the  PEO  interviews  conducted  as  part  of  the  ASSIP 
data  gathering  effort.  One  of  the  most  important  next  steps  is  to  analyze  these  results,  along 
with  the  results  of  the  other  data  gathering  efforts  discussed  in  Section  1.1  of  this  document, 
to  discover  which  problems  are  the  most  troublesome  systemically.  Identification  of  those 
problems  would  then  point  to  potential  high-impact  improvement  opportunities.  The  analysis 
task  is  already  underway  and  upon  completion,  the  SEI  will  publish  a  separate  report  discuss¬ 
ing  these  data  and  their  implications. 

The  ASSIP  Strategic  Software  Improvement  Master  Plan  (SSIMP)  for  fiscal  year  2005,  ap¬ 
proved  by  the  PEOs  and  signed  by  the  MILDEP,  outlines  several  improvement  initiatives 
developed  by  the  AAG  to  tackle  some  of  the  perceived  problems  in  SIS  acquisition.  The 
PEO  interview  results  have  helped  to  guide  those  initiatives,  which  are  discussed  briefly  in 
the  following  paragraphs. 

In  the  area  of  Policy,  one  of  the  ASSIP  initiatives  for  FY05  is  to  gather  more  details  about  the 
policy  problems.  With  the  exception  of  software  blocking  and  interoperability,  policy  com¬ 
plaints  during  the  interviews  were  non-specific.  Beginning  with  software  blocking,  the  SEI 
will  evaluate  the  policy,  its  interrelationship  with  other  policies,  and  the  complaints  levied 
against  it.  This  evaluation  will  determine  how  the  policy  might  need  to  change  to  make  it 
more  efficacious.  In  a  separate  ASSIP  initiative,  the  SEI  is  probing  interoperability  issues 
within  a  series  of  workshops  designed  to  surface  and  address  issues  related  to  implementation 
of  Army  interoperability  requirements. 

In  the  area  of  Skills  and  Training,  there  are  three  ASSIP  initiatives.  In  the  first,  the  ASSIP  is 
sponsoring  a  cohort  of  approximately  20  students  at  the  Naval  Postgraduate  School  (NPS). 
The  goal  is  to  provide  a  formal  educational  opportunity  for  PMO  and  SEC  personnel  who  are 
involved  with  SIS  acquisition.  Students  who  successfully  complete  the  4-course  sequence 
receive  a  certificate  in  DoD  Software  Improvement.  Representatives  from  NPS  and  PEO 
GCS  are  jointly  coordinating  the  effort.  In  the  second  initiative,  the  ASSIP  is  continuing  to 
sponsor  a  special  offering  of  the  SEI’s  software  architecture  curriculum.  This  special  offer¬ 
ing,  available  to  Army  PEO,  PM,  and  SEC  personnel,  will  train  a  cadre  of  software  architec¬ 
ture  professionals.  Once  trained,  these  professionals  will  then  be  able  to  apply  their  knowl¬ 
edge  to  support  the  needs  of  the  Army  acquisition  community.  In  the  third  ASSIP  education 
initiative,  the  SEI,  AAG  members,  and  DAU  are  working  to  evaluate  the  ongoing  training 
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needs  of  the  Army,  leading  to  the  development  of  a  summary  of  those  needs  mapped  to  avail¬ 
able  courses. 

In  the  Metrics  area,  another  ASSIP  initiative  focuses  on  encouraging  PMs  to  use  metrics  sys¬ 
tematically  to  understand  and  manage  their  SIS  acquisition  programs.  To  facilitate  achieving 
this  goal,  the  SEI  will  evaluate  the  effectiveness  of  metrics  used  at  the  PEO  level  and  above. 
The  SEI  will  also  recommend  an  addition  to  the  ASA(ALT)  Probability  of  Success  metric  to 
ensure  adequate  visibility  into  software  development  progress  and  risk  for  SIS  acquisitions. 
Additionally,  the  SEI  will  use  workshops  to  help  identify  the  objectives  and  information 
needs  of  Army  acquisition  managers  at  all  levels. 

Finally,  as  the  ASSIP  continues  to  promote  acquisition  process  changes,  the  SEI  will  revisit 
the  PEO  interviews  to  determine  how  those  changes  have  affected  Army  SIS  acquisition. 

The  SEI  will  also  broaden  the  interview  process  to  include  other  senior  stakeholders  in  Army 
acquisition. 
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5  Summary 


The  results  of  the  PEO  interviews  suggest  that  there  are  several  avenues  of  pursuit  for  im¬ 
proving  Army  acquisition,  especially  as  it  relates  to  software-intensive  systems.  While  each 
PEO  had  concerns  specifically  related  to  his  or  her  own  domain,  a  number  of  common  issues 
arose.  Common  issues  revolved  around  skills  and  training  of  PMs  (and  PEOs),  policy  and  its 
application,  the  acquisition  organization  and  its  workings,  use  of  metrics,  processes  and  their 
effects,  use  of  COTS  products,  and  availability  of  facilities  and  tools. 

While  the  PEOs  expressed  a  great  many  concerns  and  frustrations  with  the  current  state  of 
Army  SIS  acquisition,  they  also  offered  some  thought-provoking  ideas  for  improvement  that 
bear  repeating  here. 


With  regard  to  conflicting  policies,  a  seemingly  simple  suggestion  was  to  have 
OUSD(AT&L)  policy  writers  coordinate  with  their  service  counterparts  to  provide  consistent 
acquisition  policies  throughout  DoD. 

To  combat  the  problem  of  inconsistent  and  uncoordinated  guidance,  there  were  two  sugges¬ 
tions  worth  noting: 


•  For  each  approver,  designate  one  (and  only  one)  representative  with  whom  the 
PEOs/PMs  work. 

•  Shorten  approval  cycles  by  restructuring  the  leadership  group  to  ensure  only  decision 
makers  are  in  the  approval  loop. 

The  issue  of  command  continuity  also  generated  two  thoughts: 

•  Develop  a  “scorecard”  for  the  life  of  a  program,  so  that  a  PM  would  continue  to  bear 
some  responsibility  for  program  success  even  after  moving  to  a  new  assignment. 


•  Make  assignment  durations  flexible,  so  that  a  PM  remains  in  a  position  until  achiev¬ 
ing  some  measurable  accomplishment. 
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Several  ideas  regarding  the  PM  selection  process  were  identified: 


•  Allow  the  PM  candidates  to  bid  for  positions  that  make  the  most  sense;  that  is,  make 
the  applicant’s  preference  statement  meaningful  by  giving  due  consideration.  Focus 
on  getting  the  right  person  for  the  job. 

•  Let  the  candidates  themselves  make  the  selections,  working  together  to  go  through 
the  list,  select  the  jobs  they  want  (and  for  which  they  possess  the  right  qualifications), 
and  then  negotiate  among  themselves  to  determine  final  assignments. 

•  Do  not  put  out  “take  it  or  leave  it”  job  assignments,  which  force  too  many  PMs  to 
end  their  Army  careers. 


A  number  of  PEOs  thought  that  the  ASSIP  could  be  helpful  in  improving  acquisition  of  SIS. 
Others  were  less  optimistic,  having  seen  many  attempts  to  improve  acquisition  come  and  go 
in  the  past.  One  PEO  advised  of  the  need  to  target  “big”  changes  (>25%).  “We  can’t  meas¬ 
ure  well  enough  to  accurately  detect  1%  - 10%  change.”  Another  cautioned  against  dictating 
processes  across  the  board  stating,  “One  size  does  not  fit  all.” 

More  work  remains  to  understand  the  root  causes  of  problems  with  SIS  acquisition,  and  tack¬ 
ling  the  problems  requires  long-term  commitment.  The  ASSIP  provides  the  impetus  and  in¬ 
frastructure  to  identify  issues  and  formulate  strategic  approaches  to  address  them. 
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Feedback 


Through  its  Acquisition  Support  Program,  the  SEI  is  working  to  help  improve  SIS  acquisition 
across  the  U.S.  government.  Consequently,  the  SEI  is  very  interested  in  hearing  how  the  per¬ 
spectives  presented  here  compare  with  those  of  senior  acquisition  officials  in  the  other  ser¬ 
vices,  defense  agencies,  and  other  federal  agencies. 

Please  send  questions  or  comments  about  this  report  to  Stephen  Blanchette,  Jr. 
(sblanche@sei.cmu.edu). 


CMU/SEI-2005-SR-002 


25 


26 


CMU/SEI-2005-SR-002 


Appendix  A  Acronyms  and  Abbreviations 


The  alphabetical  listing  below  contains  all  acronyms,  abbreviations,  and  their  meanings  as 
used  in  this  report. 


AAE 

AAG 

ACAT 

AMC 

Ammo 

ASA(ALT) 

ASD(NII) 

ASMD 

ASSIP 

AVN 

BFI 

C3T 

C4ISR 


Army  Acquisition  Executive 
ASSIP  Action  Group 
Acquisition  Category 
Army  Materiel  Command 
Ammunition 

Assistant  Secretary  of  the  Army  for  Acquisition,  Logistics,  and  Technol¬ 
ogy 

Assistant  Secretary  of  Defense  for  Networks  and  Information  Integration 

Air,  Space  and  Missile  Defense 

Army  Strategic  Software  Improvement  Program 

Aviation 

Benchmarking  for  Improvement 

Command,  Control  and  Communications  Tactical 

Command  Control,  Communications,  Computers,  Intelligence,  Surveil¬ 
lance,  and  Reconnaissance 
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CBD 

CMM 

CMMI 

CMU 

COTS 

CS&CSS 

DA 

DAU 

DoD 

DSB 

EIS 

FBCB2 

FCS 

FY 

GAO 

GCS 

HQDA 

IEPR 


Chemical  and  Biological  Defense 

Capability  Maturity  Model 

Capability  Maturity  Model  Integration 

Carnegie  Mellon  University 

Commercial  Off-the-Shelf 

Combat  Support  and  Combat  Service  Support 

Department  of  the  Army 

Defense  Acquisition  University 

Department  of  Defense 

Defense  Science  Board 

Enterprise  Information  Systems 

Force  XXI  Battle  Command  Brigade  and  Below 

Future  Combat  Systems 

Fiscal  Year 

Government  Accounting  Office 
Ground  Combat  Systems 
Headquarters,  Department  of  the  Army 
Independent  Expert  Program  Review 
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DBW&S 

IPT 

JPEO 

JPO 

JTRS 

LSI 

LTG 

MDA 

NPS 

OSD 

OUSD(AT&L) 

PEO 

PM 


PMO 


Intelligence,  Electronic  Warfare  and  Sensors 

Integrated  Product  Team 

Joint  Program  Executive  Officer 

Joint  Program  Executive  Office 

Joint  Program  Office 

Joint  Tactical  Radio  System 

Lead  System  Integrator 

Lieutenant  General 

Milestone  Decision  Authority 

Naval  Postgraduate  School 

Office  of  the  Secretary  of  Defense 

Office  of  the  Undersecretary  of  Defense  for  Acquisition,  Technology,  and 
Logistics 

Program  Executive  Officer 
Program  Executive  Office 
Project  Manager 
Program  Manager 
Product  Manager 
Program  Management  Office 
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PPSS 


Post-Production  Software  Support 


RDECOM 

SAIP 

SCAMPI 

SEC 

SEI 

SIS 

SR 

SSG 

SSIMP 

STRI 

SW 

TAI 

TM 

TRADOC 

TSPR 

UA 

USA 


Research,  Development,  and  Engineering  Command 

Software  Acquisition  Improvement  Plan 

Standard  CMMI  Appraisal  Method  for  Process  Improvement 

Software  Engineering  Center 

Software  Engineering  Institute 

Software-Intensive  Systems 

Special  Report 

Senior  Steering  Group 

Strategic  Software  Improvement  Master  Plan 
Simulation,  Training,  and  Instrumentation 
Software 

Tri-Service  Assessment  Initiative 

Tactical  Missiles 

Training  and  Doctrine  Command 

Total  System  Performance  Responsibility 

Unit  of  Action 

United  States  Army 
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USAAA 

USAF 


United  States  Army  Audit  Agency 
United  States  Air  Force 
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Appendix  B  Interview  Questions 


Primary  Questions 

The  primary  questions  were  asked  of  all  PEOs  interviewed. 


General 


1.  How  long  have  you  been  in  your  current  position? 

2.  What  are  your  primary  responsibilities? 


Perceived  Problems  &  Risks 


The  ASA(ALT)  is  concerned  that  we  are  addressing  the  right  problems  and  identifying  what 
is  needed  by  the  community. 

3.  What  do  you  see  as  your  primary  problems  or  needs  with  acquisition  of  software¬ 
intensive  systems? 

4.  With  respect  to  your  identified  problems  or  risks,  how  do  you  believe  the  ASSIP 
could  help? 


Perceived  Problems  &  Risks,  continued 


[Note:  This  question  refers  to  the  chart  in  Figure  3.] 

The  challenge  areas  listed  along  the  top  row  of  the  chart  were  indicated  by  the  results  of  vari¬ 
ous  data-gathering  efforts  that  are  represented  along  the  left  column.  Indicate  the  degree  of 
your  concurrence  with  these  findings  by  marking  the  appropriate  number  in  the  cell  along  the 
bottom  row  of  the  chart. 

5.  For  the  indicated  challenge  areas,  what  are  your  suggestions  for  how  these  can  be 
addressed? 
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Army  Acquisition  Challenge  Areas 


Project 
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Figure  3:  Army  Acquisition  Challenge  Areas 


Leadership 


We  are  building  a  Leadership  Awareness  Briefing.  The  audience  for  this  briefing  is  the  Army 
senior  acquisition  executive  level  (including  both  PEOs  and  DA/OSD  levels).The  purpose  of 
this  briefing  is  to  raise  leadership  awareness  of  the  challenges  involved  with  the  acquisition 
of  software-intensive  systems,  so  that  they  can  become  part  of  the  solution  and  not  part  of  the 
problem. 

6.  What  are  the  key  points  that  you  think  a  senior  leader  needs  to  understand  to  make 
your  job  (and  your  PMs’ job)  successful? 


Metrics  and  Reporting  Information  '  ’  'H'/’CA* " ‘»v 


Leading  software  development  companies  have  disciplined  development  processes  and  use 
metrics  methodically  to  ensure  that  software  is  developed  within  cost,  schedule,  and  perform- 
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ance  targets.  To  track  progress,  confirm  knowledge,  manage  risk,  improve  processes,  and  en¬ 
sure  that  customers  are  well  informed,  leading  developers  collect  metrics  that  address: 

•  cost  •  test 

•  schedule  •  defects 

•  size  •  quality 

•  requirements 

ASSIP  will  be  pulling  together  a  common  picture  of  how  measurement  can  be  implemented 
by  PMs  to  manage  their  programs  and  to  report  status  to  the  PEOs.  The  vision  is  to  define  a 
set  of  metrics  that  PEOs  and  DA/OSD  can  use  for  oversight  and  for  program  insights. 

7.  What  measures  are  your  PMs  using? 

8.  What  measures  do  you  look  at  each  month  and  how  do  you  use  them? 

9.  What  measures  do  you  report  up  the  chain  to  DA  and  OSD? 

10.  What  other  types  of  information  would  you  like  to  have? 


Skill  Competencies  and  Training 


11.  To  what  extent  do  your  PMs  rely  on  the  expertise  of  the  RDECOM  Software  Engi¬ 
neering  Centers? 


Program-Specific  Improvement  Planning 


The  ASSIP  and  Section  804  are  directed  to  improve  acquisition  of  software  intensive  sys¬ 
tems — both  process  and  products.  As  you  know,  the  ASSIP,  FY04  SSIMP  includes  the  fol¬ 
lowing: 

•  Implementation  of  a  Software  Acquisition  Improvement  Plan  (SALP). 

•  The  AAG  will  help  shape  and  coordinate  the  SAIPs  and  the  FY05  SSIMP.  As  occurred 
in  the  creation  of  the  FY04  SSIMP,  the  PEO  will  be  asked  by  the  MILDEP  to  support  it. 


12.  What  are  your  thoughts  and  opinions  as  to  these  expectations? 
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[Note:  If  the  secondary  questions  were  included  in  an  interview,  they  were  asked  before  this 
question.] 

13.  Is  there  any  other  guidance  you  wish  to  offer?  How  can  the  ASSIP  be  effective  in 
helping  you  and  your  PMs? 
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Secondary  Questions 

The  secondary  questions  were  asked  selectively,  and  only  if  time  permitted.  Some  of  them 
were  specifically  targeted  to  provide  guidance  for  execution  of  the  ASSIP  rather  than  to  de¬ 
termine  opinions  about  Army  acquisition  in  general. 


Future  Force  Workshop 


As  the  MILDEP,  LTG  Caldwell9  asked,  “What  can  the  ASSIP  do  to  help  the  Future  Force?” 

In  response  to  this  question,  there  has  been  a  suggestion  to  hold  a  workshop  for  principal  pro¬ 
grams  to  ascertain  if  they  are  on  the  right  path,  regarding  software,  to  reach  the  Future  Force 
together. 


Such  a  workshop  could  focus  on  identifying  what  we  are  doing  right  across  our  programs, 
and  also  what  is  not  being  addressed  that  needs  to  be  addressed,  in  order  to  achieve  better 
coordination,  cooperation  and  synchronization  (recognizing  that  there  are  numerous  IPTs  and 
other  groups  associated  with  interoperability,  FCS  initiatives  and  Software  Blocking,  etc.). 

14.  Is  this  the  correct  focus  for  such  a  workshop? 

There  have  been  topics  identified  as  being  important  for  such  a  workshop.  Among  these  top¬ 
ics  are: 

•  SW  architecture 

•  Interoperability 

•  General  programmatic  issues 

15.  What  is  your  opinion  of  including  these  topics? 

16.  What  other  topics  would  you  suggest  for  such  a  workshop? 

17.  Do  you  have  any  other  suggestions  regarding  the  workshop? 


9  Lieutenant  General  Caldwell  was  the  military  deputy  to  the  ASA(ALT)  until  his  retirement  in 
January  2004. 
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The  following  three  statements  are  one  PEO’s  remarks  about  the  FY04  SSIMP.  Please  com¬ 
ment  and  offer  a  course  of  action  on  these  statements: 

18.  “Our  biggest  concern  is  with  the  acquisition  process,  specifically  the  requirements 
approval  to  permit  the  flexibility  needed  to  design  systems  for  incremental  change.” 

19.  “There  needs  to  he  significant  reduction  in  oversight  by  offices  and  agencies  which 
should  have  provided  input  to  the  original  requirements.” 

20.  “My  expectation  would  be  that  we  need  the  ASSIPto  put  dollars  against  ‘blueprint¬ 
ing’  the  oversight  and  review  process  as  well  as  clarifying  the  approval  process  for 
requirements  and  change  orders.” 


21.  What  changes  in  Army  acquisition  policy  do  you  think  could  help  you  and  your 
PMs? 


Skill  Competencies  and  Training 


22.  How  well  is  your  staff  trained  with  respect  to  Systems  Engineering  and  Software 
Engineering? 


23.  Do  you  believe  your  PMs  give  the  appropriate  priority  to  acquisition  training? 


24.  Are  there  training  plans  for  your  PMs  and  PMO  personnel? 

25.  What  is  your  perspective  on  the  Software  Engineering  Institute’s  (SEI’s)  suite  of 
process  improvement  assets  such  as  CMMI,  CMMI-Acquisition  Module,  SCAMPI, 
etc.? 
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Appendix  C  Selected  Demographics 


The  tables  and  charts  below  display  selected  demographic  information  to  help  the  audience 
understand  the  interview  results. 

Interviewees 

Table  2  lists  the  officers  and  senior  executives10  interviewed  for  this  study. 


Table  2:  Interviewees 


Name 

Title(s)  1 

Mr.  Edward  T.  Baer 

•  Program  Executive  Officer  for  Intelligence,  Electronic  Warfare,  and 

Sensors  (PEO  IEW&S) 

Dr.  James  T.  Blake 

•  Acting  Program  Executive  Officer  for  Simulation,  Training,  and  In¬ 
strumentation  (PEO  STRI) 

Mr.  Paul  Bogosian 

•  Program  Executive  Officer  for  Aviation  (PEO  AVN) 

Mr.  Kevin  Carroll 

•  Program  Executive  Officer  for  Enterprise  Information  Systems  (PEO 

EIS) 

Brigadier  General  Paul  S.  Izzo, 

USA 

•  Commanding  General,  Picatinny  Arsenal 

•  Program  Executive  Officer  for  Ammunition  (PEO  Ammo) 

Colonel  Steven  MacLaird,  USAF 

•  Program  Director,  Joint  Tactical  Radio  System  Joint  Program  Office 

(JTRS  JPO) 

Major  General  Michael  R. 

Mazzucchi,  USA 

•  Commanding  General,  United  States  Army  Communications- 

Electronics  Command  and  Fort  Monmouth 

•  Program  Executive  Officer  for  Command,  Control,  and  Communica¬ 
tions  Tactical  (PEO  C3T) 

Brigadier  General  James  R.  Moran, 

USA 

•  Commanding  General,  Soldier  Systems  Center 

•  Deputy  Commanding  General  for  Operations.  United  States  Army  Re¬ 
search,  Development  and  Engineering  Command 

•  Program  Executive  Officer  for  Soldier  (PEO  Soldier) 

10  All  ranks,  titles,  and  commands  are  as  of  the  time  of  the  respective  interviews. 
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Name 

Title(s)  1 

Dr.  Shelba  Proffitt 

•  Deputy  Program  Executive  Officer  for  Air,  Space  and  Missile  Defense 

(PEO  ASMD) 

Brigadier  General  Stephen  V. 

•  Joint  Program  Executive  Officer  for  Chemical  and  Biological  Defense 

Reeves,  USA 

(JPEO  CBD) 

As  one  would  expect,  the  ongoing  contingency  operations  in  southwest  Asia  keep  the  PEOs 
very  busy  addressing  soldiers’  needs  in  the  field.  As  a  result,  PEO  CS&CSS,  PEO  GCS,  and 
PEO  TM  were  unable  to  schedule  interview  time.  PM  Unit  of  Action  (UA)  was  unavailable 
due  to  the  acceleration  of  the  FCS  program  and  resulting  replanning  efforts.  In  addition,  the 
PEOs  for  ASMD  and  STRI  were  on  special  assignments  in  Iraq,  so  their  deputies  handled  the 
interviews  instead. 

Time  in  Position 

Although  each  interviewee  is  a  senior  acquisition  professional  with  many  years  of  experience 
in  lower  level  positions,  not  all  of  them  possess  equivalent  experience  in  their  current  roles. 
The  time  that  an  interviewee  held  the  position  at  the  time  of  the  interview  varied,  from  a  little 
over  a  year  to  several  years.  Figure  4  shows  each  interviewee’s  tenure.  Further,  Figure  5 
shows  the  summary  statistics  for  number  of  years  on  the  job.  Of  the  five  interviewees  with 
more  than  three  years  of  service  in  their  current  positions,  three  are  civilians. 


Respondent 


Figure  4:  Number  of  Years  in  Position  (at  time  of  interview) 


40 


CMU/SEI-2005-SR-002 


Figure  5:  Summary  -  Number  of  Years  in  Position 


Grades 

Figure  6  shows  the  distribution  of  grades  among  the  interviewees.  For  comparison,  Figure  7 
shows  the  overall  distribution  of  grades  for  the  PEOs  themselves.  While  Brigadier  General  is 
the  predominant  rank  among  all  PEOs,  the  recent  trend  has  been  to  assign  more  civilians  to 
the  position.  Civilian  senior  executives  now  make  up  nearly  one-third  of  the  community 
overall.  The  interview  population  included  two  deputies,  which  skews  the  distribution,  mak¬ 
ing  it  50  percent  civilians. 
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Figure  6:  Grade  Distribution  Among  Interviewees 


Grade  Distribution  Among  PEOs 


■  Major  General 

■  Brigadier  General 

1  ■  Colonel 

B  Senior  Executive 


Figure  7:  Grade  Distribution  Among  All  PEOs 
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